Pattern Recognition and Feed-forward Networks

A feed-forward network can be viewed as a graphical representation of parametric function which takes a set of input values and maps them to a corresponding set of output values (Bishop, 1995). Figure 1 shows an example of a feed-forward network of a kind that is widely used in practical applications. Nodes in the outputs hidden units inputs bias bias x 0 x 1 x d y 1 z 1 z 0 y c z M Figure 1: A feed-forward network having two layers of adaptive parameters. graph represent either inputs, outputs or 'hidden' variables, while the edges of the graph correspond to the adaptive parameters. We can write down the analytic function corresponding to this network follows. The output of the jth hidden node is obtained by first forming a weighted linear combination of the d input values x i to give a j = d i=1 u ji x i + b j. The value of hidden variable j is then obtained by transforming the linear sum in (1) using an activation function g(·) to give z j = g(a j). (2) Finally, the outputs of the network are obtained by forming linear combinations of the hidden variables to give a k = M j=1 v kj z j + c k. The parameters {u ji , v kj } are called weights while {b j , c k } are called biases, and together they constitute the adaptive parameters in the network. There is a one-to-one correspondence between the variables and parameters in the analytic function and the nodes and edges respectively in the graph.